{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "75f9200e-7830-4ee5-8637-e67b5df57eac",
   "metadata": {},
   "source": [
    "# Intel PyTorch GPU Training and Inference with AMP"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48eb565f-ef03-40cb-9182-5b2b752331e8",
   "metadata": {},
   "source": [
    "The `PyTorch Training Optimizations with Advanced Matrix Extensions Bfloat16` sample will demonstrate how to train a ResNet50 model using the CIFAR10 dataset using the Intel® Extension for PyTorch*.\n",
    "\n",
    "The Intel® Extension for PyTorch* extends PyTorch* with optimizations for extra performance boost on Intel® hardware. While most of the optimizations will be included in future PyTorch* releases, the extension delivers up-to-date features and optimizations for PyTorch on Intel® hardware. For example, newer optimizations include AVX-512 Vector Neural Network Instructions (AVX512 VNNI) and Intel® Advanced Matrix Extensions (Intel® AMX).\n",
    "\n",
    "| Area                  | Description\n",
    "|:---                   |:---\n",
    "| What you will learn   | Training performance improvements using Intel® Extension for PyTorch* with Intel® AMX BF16\n",
    "| Time to complete      | 20 minutes\n",
    "| Category              | Code Optimization\n",
    "\n",
    "## Purpose\n",
    "\n",
    "The Intel® Extension for PyTorch* gives users the ability to speed up training on Intel® Xeon Scalable processors with lower precision data formats and specialized computer instructions. The bfloat16 (BF16) data format uses half the bit width of floating-point-32 (FP32), lowering the amount of memory needed and execution time to process. You should notice performance optimization with the AMX instruction set when compared to AVX-512.\n",
    "\n",
    "## Prerequisites\n",
    "\n",
    "| Optimized for           | Description\n",
    "|:---                     |:---\n",
    "| OS                      | Ubuntu* 18.04 or newer\n",
    "| Hardware                | 4th Gen Intel® Xeon® Scalable Processors or newer\n",
    "| Software                | Intel® Extension for PyTorch*\n",
    "\n",
    "## Key Implementation Details\n",
    "\n",
    "This code sample will train a ResNet50 model using the CIFAR10 dataset while using Intel® Extension for PyTorch*. The model is trained using FP32 and BF16 precision, including the use of Intel® Advanced Matrix Extensions (AMX) on BF16. AMX is supported on BF16 and INT8 data types starting with the 4th Generation of Xeon Scalable Processors. The training time will be compared, showcasing the speedup of BF16 and AMX.\n",
    "\n",
    ">**Note**: Training is not performed using INT8 since using a lower precision will train a model with fewer parameters, which is likely to underfit and not generalize well."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e41ce52-c94c-4bdf-a528-0e0200fd5501",
   "metadata": {},
   "source": [
    "## Installation of required packages\n",
    "\n",
    "Ensure the kernel is set to Pytorch-GPU before running the following code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b27f39c2-6d92-40cf-a878-b2fd2e7c818b",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install matplotlib requests tqdm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e4eedf0-5c7c-49d3-be15-f46b4988d9ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from time import time\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import torch\n",
    "import torchvision\n",
    "import intel_extension_for_pytorch as ipex\n",
    "from tqdm import tqdm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17246f67-0059-4b5f-afe8-a105d767b139",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Hyperparameters and constants\n",
    "LR = 0.01\n",
    "MOMENTUM = 0.9\n",
    "DATA = 'datasets/cifar10/'\n",
    "epochs=1\n",
    "batch_size=128"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd146688",
   "metadata": {},
   "source": [
    "### Check for env setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2917da44",
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.xpu.is_available()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e883a39",
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "  device = \"xpu\" if torch.xpu.is_available() else \"cpu\" \n",
    "  \n",
    "except:\n",
    "  device = \"cpu\"  \n",
    "\n",
    "if device == \"xpu\": # Intel dGPU is recognized as device type xpu\n",
    "  print(\"IPEX_XPU is present and Intel GPU is available to use for PyTorch\")\n",
    "  device = \"gpu\"\n",
    "else:\n",
    "  print(\"using CPU device for PyTorch\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ab47ca1-d8fd-4e69-bca6-cb29dba6596b",
   "metadata": {},
   "source": [
    "## Loading the dataset\n",
    "The CIFAR10 dataset is used for this sample. Dataset is being downloaded from built-in datasets available in the torchvision.datasets module. Batch size will be set to 128."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c3d9563-e40e-48b4-a311-a71a762b5b58",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Dataloader operations\n",
    "transform = torchvision.transforms.Compose([\n",
    "torchvision.transforms.Resize((224, 224)),\n",
    "torchvision.transforms.ToTensor(),\n",
    "torchvision.transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))\n",
    "])\n",
    "train_dataset = torchvision.datasets.CIFAR10(\n",
    "        root=DATA,\n",
    "        train = True,\n",
    "        transform=transform,\n",
    "        download=True,\n",
    ")\n",
    "train_loader = torch.utils.data.DataLoader(\n",
    "        dataset=train_dataset,\n",
    "        batch_size=batch_size\n",
    ")\n",
    "\n",
    "test_dataset = torchvision.datasets.CIFAR10(root=DATA, train = False,\n",
    "                                       download=True, transform=transform)\n",
    "test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=batch_size )\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ccd66ee-aac5-4a60-8f66-417612d4d3af",
   "metadata": {},
   "source": [
    "## Training the Model\n",
    "The function below will train the ResNet50 model based on whether it should use CPU or Intel dGPU, and whether to use FP32 or BF16 data type. To use Intel dGPU, we need to transfer model and data to xpu device using `to(\"xpu\")`.To use BF16 in operations on CPU, use the `torch.cpu.amp.autocast()` function to perform forward and backward propagation.\n",
    "\n",
    "For Intel dGPU, `torch.xpu.amp` provides convenience for auto data type conversion at runtime, allowing deep learning workloads to benefit from lower-precision floating point data types like `torch.float16` or `torch.bfloat16`, which offer lighter calculation workload and smaller memory usage. However, lower-precision data types sacrifice accuracy for performance. The Auto Mixed Precision (AMP) feature automates data type conversions for operators, allowing for a trade-off between accuracy and performance. `torch.xpu.amp.autocast` is a context manager that enables scopes of the script to run with mixed precision, where operations are performed in a data type chosen by the autocast class to improve performance while maintaining accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b8e21c9-aaa5-4f75-b00a-0d875cc0bfba",
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Function to run a test case\n",
    "\"\"\"\n",
    "def trainModel(train_loader, modelName=\"myModel\", device=\"cpu\", dataType=\"fp32\"):\n",
    "    \"\"\"\n",
    "    Input parameters\n",
    "        train_loader: a torch DataLoader object containing the training data with images and labels\n",
    "        modelName: a string representing the name of the model\n",
    "        device: the device to use - cpu or gpu\n",
    "        dataType: the data type for model parameters, supported values - fp32, bf16\n",
    "    Return value\n",
    "        training_time: the time in seconds it takes to train the model\n",
    "    \"\"\"\n",
    "\n",
    "    # Initialize the model and add a fully connected layer for finetuning the model on CIFAR dataset(with 10 classes). Originally, the ResNet50 is trained with ImageNet dataset(1000 classes)   \n",
    "    model = torchvision.models.resnet50(pretrained=True)\n",
    "    model.fc = torch.nn.Linear(2048,10)\n",
    "    lin_layer = model.fc\n",
    "    new_layer = torch.nn.Sequential(\n",
    "        lin_layer,\n",
    "        torch.nn.Softmax(dim=1)\n",
    "    )\n",
    "    model.fc = new_layer\n",
    "\n",
    "    #Define loss function and optimization methodology\n",
    "    criterion = torch.nn.CrossEntropyLoss()\n",
    "    optimizer = torch.optim.SGD(model.parameters(), lr=LR, momentum=MOMENTUM)\n",
    "    model.train()\n",
    "\n",
    "    #export model and criterian to XPU device. GPU specific code\n",
    "    if device == \"gpu\":\n",
    "        model = model.to(\"xpu:0\") ## if we have two Intel dGPU device, we can specify xpu:0 or xpu:1\n",
    "        criterion = criterion.to(\"xpu:0\") \n",
    "\n",
    "    #Optimize with BF16 or FP32(default) . BF16 specific code\n",
    "    if \"bf16\" == dataType:\n",
    "        model, optimizer = ipex.optimize(model, optimizer=optimizer, dtype=torch.bfloat16)\n",
    "    else:\n",
    "        model, optimizer = ipex.optimize(model, optimizer=optimizer, dtype=torch.float32)\n",
    "\n",
    "    #Train the model\n",
    "    num_batches = len(train_loader) * epochs\n",
    "    \n",
    "\n",
    "    for i in range(epochs):\n",
    "        running_loss = 0.0\n",
    "\n",
    "        for batch_idx, (data, target) in enumerate(train_loader):\n",
    "            optimizer.zero_grad()\n",
    "            # export data to XPU device. GPU specific code\n",
    "            if device == \"gpu\":\n",
    "                data = data.to(\"xpu:0\")\n",
    "                target = target.to(\"xpu:0\")\n",
    "\n",
    "            # Apply Auto-mixed precision(BF16)  \n",
    "            if \"bf16\" == dataType:\n",
    "                with torch.xpu.amp.autocast(enabled=True, dtype=torch.bfloat16):\n",
    "\n",
    "                    output = model(data)\n",
    "                    loss = criterion(output, target)\n",
    "                    loss.backward()\n",
    "                    optimizer.step()\n",
    "                    running_loss += loss.item()\n",
    "\n",
    "            else:\n",
    "\n",
    "                output = model(data)\n",
    "                loss = criterion(output, target)\n",
    "                loss.backward()\n",
    "                optimizer.step()\n",
    "                running_loss += loss.item()\n",
    "\n",
    "\n",
    "            # Showing Average loss after 50 batches\n",
    "            if 0 == (batch_idx+1) % 50:\n",
    "                print(\"Batch %d/%d complete\" %(batch_idx+1, num_batches))\n",
    "                print(f' average loss: {running_loss / 50:.3f}')\n",
    "                running_loss = 0.0\n",
    "\n",
    "    # Save a checkpoint of the trained model\n",
    "    torch.save({\n",
    "        'model_state_dict': model.state_dict(),\n",
    "        'optimizer_state_dict': optimizer.state_dict(),\n",
    "        }, 'checkpoint_%s.pth' %modelName)\n",
    "    print(f'\\n Training finished and model is saved as checkpoint_{modelName}.pth')\n",
    "    return None\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d5c2307-0547-4e81-bbcc-25fc09a830a0",
   "metadata": {},
   "source": [
    "### Model Training with default FP32 precision(Recommended for inference comparison)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9680926b",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Model Training\n",
    "print(\"Training model with FP32 on GPU, will be saved as checkpoint_gpu_rn50.pth\")\n",
    "trainModel(train_loader, modelName=\"gpu_rn50\", device=\"gpu\", dataType=\"fp32\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d6d8fb4-983b-417f-88c3-dc2747de3d4a",
   "metadata": {},
   "source": [
    "### Model Training with default AMP BF16(Optional) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1ff86b79-9bc8-4e6e-8d43-b2cfa1eb500f",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Model Training\n",
    "print(\"Training model with BF16 on GPU, will be saved as checkpoint_gpu_rn50_bf16.pth\")\n",
    "trainModel(train_loader, modelName=\"gpu_rn50_bf16\", device=\"gpu\", dataType=\"bf16\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76698bac",
   "metadata": {},
   "source": [
    "## FP32 & AMP BF16 Model Evaluation if trained with FP32 precision"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "798f923b",
   "metadata": {},
   "source": [
    "### Load model from saved model file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cff1ce45",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Load model structure from torchvision and weights from saved checkpoint file\n",
    "def load_model(cp_file = 'checkpoint_rn50.pth'):\n",
    "    model = torchvision.models.resnet50()\n",
    "    model.fc = torch.nn.Linear(2048,10)\n",
    "    lin_layer = model.fc\n",
    "    new_layer = torch.nn.Sequential(\n",
    "        lin_layer,\n",
    "        torch.nn.Softmax(dim=1)\n",
    "    )\n",
    "    model.fc = new_layer\n",
    "\n",
    "    checkpoint = torch.load(cp_file)\n",
    "    model.load_state_dict(checkpoint['model_state_dict']) \n",
    "    return model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2504363-367c-4afd-9e49-b5f3a1af4367",
   "metadata": {},
   "source": [
    "### Applying Intel® Extension for PyTorch (IPEX) optimizations and Converting model to TorchScript(Optional)\n",
    "TorchScript is a way to create serializable and optimizable models from PyTorch code. Any code written in TorchScript can be saved from your Python process and loaded in a process where there is no Python dependency. `torch.jit.trace` and `torch.jit.freeze` is used for converting the model to TorchScript. `torch.jit.trace` will trace a function and return an executable or ScriptFunction that will be optimized using just-in-time compilation. `torch.jit.freeze` will clone executable or ScriptFunction and attempt to inline the cloned module's submodules, parameters, and attributes as constants in the TorchScript IR Graph.\n",
    "\n",
    "Intel® Extension for PyTorch (IPEX) provides optimizations for both eager mode and graph mode, however, compared to eager mode, graph mode in PyTorch normally yields better performance from optimization techniques such as operation fusion, and Intel® Extension for PyTorch (IPEX) amplified them with more comprehensive graph optimizations. Therefore we recommended you to take advantage of Intel® Extension for PyTorch (IPEX) with TorchScript. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "721824ad-4827-4a29-9c98-15ef92d0f541",
   "metadata": {},
   "outputs": [],
   "source": [
    "def ipex_jit_optimize(model, dataType = \"fp32\" , device=\"cpu\"):\n",
    "    model.eval()\n",
    "    if device==\"gpu\":\n",
    "        model = model.to(\"xpu:0\")\n",
    "    if dataType==\"bf16\":\n",
    "        model = ipex.optimize(model, dtype=torch.bfloat16)\n",
    "    else:\n",
    "        model = ipex.optimize(model, dtype = torch.float32)\n",
    "            \n",
    "    with torch.no_grad():\n",
    "        d = torch.rand(1, 3, 224, 224)\n",
    "        if device==\"gpu\": \n",
    "            d = d.to(\"xpu:0\")\n",
    "            \n",
    "        if dataType==\"bf16\": \n",
    "          with torch.xpu.amp.autocast(enabled=True, dtype=torch.bfloat16): \n",
    "            jit_model = torch.jit.trace(model, d) # JIT trace the optimized model\n",
    "            jit_model = torch.jit.freeze(jit_model) # JIT freeze the traced model\n",
    "        else:\n",
    "          jit_model = torch.jit.trace(model, d) # JIT trace the optimized model\n",
    "          jit_model = torch.jit.freeze(jit_model) # JIT freeze the traced model              \n",
    "    return jit_model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "581c85a0-31eb-4c8b-adbf-713c80919b05",
   "metadata": {},
   "source": [
    "### Inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "022f7cfe-2c32-40f2-8880-34a8a0ec31d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "def inferModel(model, test_loader, device=\"cpu\" , dataType='fp32'):\n",
    "    correct = 0\n",
    "    total = 0\n",
    "    if device == \"gpu\":\n",
    "        model = model.to(\"xpu:0\")\n",
    "    infer_time = 0\n",
    "\n",
    "    with torch.no_grad():\n",
    "        num_batches = len(test_loader)\n",
    "        batches=0\n",
    "                   \n",
    "        for i, data in tqdm(enumerate(test_loader)):\n",
    "            \n",
    "            # Record time for Inference\n",
    "            torch.xpu.synchronize()\n",
    "            start_time = time()\n",
    "            images, labels = data\n",
    "            if device ==\"gpu\":\n",
    "                images = images.to(\"xpu:0\")\n",
    "                 \n",
    "            outputs = model(images)\n",
    "            outputs = outputs.to(\"cpu\") # Need model outputs back to CPU(Host) again to remove Device(GPU) to Host overhead as all the accuracy related computation is going to happen on CPU\n",
    "            _, predicted = torch.max(outputs.data, 1)\n",
    "            \n",
    "            total += labels.size(0)\n",
    "            correct += (predicted == labels).sum().item()        \n",
    "            \n",
    "            # Record time after finishing batch inference\n",
    "            torch.xpu.synchronize()\n",
    "            end_time = time()      \n",
    "\n",
    "            if i>=3 and i<=num_batches-3: # Ignoring a few start and end batches for consistent and accurate latency measure \n",
    "                infer_time += (end_time-start_time)\n",
    "                batches += 1\n",
    "            #Skip last few batches     \n",
    "            if i == num_batches - 3:\n",
    "                break    \n",
    "\n",
    "    accuracy = 100 * correct / total\n",
    "    return accuracy, infer_time*1000/(batches*batch_size)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "32e9190d",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Evaluation of different models\n",
    "def Eval_model(cp_file = 'checkpoint_model.pth', dataType = \"fp32\" , device=\"gpu\" ):\n",
    "    model = load_model(cp_file)\n",
    "    model = ipex_jit_optimize(model, dataType , device)\n",
    "    accuracy, latency = inferModel(model, test_loader, device, dataType )\n",
    "    print(f' Model accuracy: {accuracy} and Average Inference latency: {latency} \\n'  )\n",
    "    return accuracy, latency"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12e13e32",
   "metadata": {},
   "source": [
    "### Accuracy and Inference latency check"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7516593",
   "metadata": {},
   "source": [
    "For FP32 model on GPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2c53b7dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "#For FP32 model on GPU\n",
    "print(\"Model evaluation with FP32 on GPU\")\n",
    "acc_fp32, fp32_avg_latency = Eval_model(cp_file = 'checkpoint_gpu_rn50.pth', dataType = \"fp32\" , device=\"gpu\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87063fd1",
   "metadata": {},
   "source": [
    "For BF16 model on GPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a59b600",
   "metadata": {},
   "outputs": [],
   "source": [
    "#For AMP BF16 model on GPU\n",
    "print(\"Model evaluation with AMP BF16 on GPU\")\n",
    "acc_bf16, bf16_avg_latency = Eval_model(cp_file = 'checkpoint_gpu_rn50.pth', dataType = \"bf16\" , device=\"gpu\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1d2e1e7-40db-4e94-9f04-0d812992169a",
   "metadata": {},
   "source": [
    "## Summary of Results for GPU\n",
    "The following cells below will summarize the training times for all three cases and display graphs to show the performance speedup."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce3aa054-a36e-47e8-a646-ac939b74776a",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Summary \n",
    "print(\"Summary\")\n",
    "print(f'Inference average latecy for FP32  on GPU is:  {fp32_avg_latency} ')\n",
    "print(f'Inference average latency for AMP BF16 on GPU is:  {bf16_avg_latency} ')\n",
    "\n",
    "speedup_from_amp_bf16 = fp32_avg_latency / bf16_avg_latency\n",
    "print(\"Inference with BF16 is %.2fX faster than FP32 on GPU\" %speedup_from_amp_bf16)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "69c69b95-3d3f-49c0-82ac-1de5bf16ce69",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure()\n",
    "plt.title(\"ResNet50 Inference Latency Comparison\")\n",
    "plt.xlabel(\"Test Case\")\n",
    "plt.ylabel(\"Inference Latency per sample(ms)\")\n",
    "plt.bar([\"FP32 on GPU\", \"AMP BF16 on GPU\"], [fp32_avg_latency, bf16_avg_latency])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "361304df",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure()\n",
    "plt.title(\"Accuracy Comparison\")\n",
    "plt.xlabel(\"Test Case\")\n",
    "plt.ylabel(\"Accuracy(%)\")\n",
    "plt.bar([\"FP32 on GPU\", \"AMP BF16 on GPU\"], [acc_fp32, acc_bf16])\n",
    "print(f'Accuracy drop with AMP BF16 is: {acc_fp32-acc_bf16}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0990591d-3f4b-40f9-85fc-6b9a7c8e2123",
   "metadata": {},
   "outputs": [],
   "source": [
    "speedup_from_bf16_on_gpu = fp32_avg_latency/bf16_avg_latency\n",
    "plt.figure()\n",
    "plt.title(\"GPU AMP BF16 Speedup\")\n",
    "plt.xlabel(\"Test Case\")\n",
    "plt.ylabel(\"SpeedUp\")\n",
    "plt.bar([\"FP32 on GPU\", \"Speed Up from AMP BF16 on GPU\"], [1, speedup_from_bf16_on_gpu])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa0877d6-e045-4091-b5e4-4dfcb6d04f7d",
   "metadata": {},
   "outputs": [],
   "source": [
    "print('[CODE_SAMPLE_COMPLETED_SUCCESFULLY]')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  },
  "vscode": {
   "interpreter": {
    "hash": "ed6ae0d06e7bec0fef5f1fb38f177ceea45508ce95c68ed2f49461dd6a888a39"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
